The present invention relates to an image processing apparatus for joining (restoring composition) a plurality of images taken to which one composition is divided with an overlap area where desired object exists or synthesizing a plurality of images taken with a different exposure, and more particularly to an image processing apparatus for extending a viewing angle of a joined image and its dynamic range of a synthesized image.
In recent years, personal computers (hereinafter called PC) got much more capability and their price is reducing in accordance with improvement of manufacture technique, so that they have been widely used in many companies, education, and home.
To input images to PC, an image is optically picked up from a film photographed by a conventional camera and is converted to an image signal to be input.
In addition to camera, imaging apparatus such as a video camera for taking the image are used in various situations. Particularly, in a digital still camera, a film, which is used in the general camera, doesn't have to be, used. Instead, the image is converted to a digital signal and recorded in storage medium, which is magnetically or optically recordable, so that the digital signal is input to the inputting device. Then, the image is reconstructed to be output to a display or a printer. As a result, developing process for films is unnecessary, and erasing and editing can be easily performed.
Moreover, demand for digital still cameras has been rapidly increased for the purpose of using the Internet communication in accordance with an increase Internet users, reduce the price of the digital still camera.
The digital still camera get a image of the object as an image signal by a solid state imaging device, such as CCD, using photoelectric conversion. However, since resolution and a dynamic range are low as compared with the ordinary film, the technique to make high resolution images and to extend a dynamic range of images have been strongly desired.
One of methods for obtaining high resolution image is increasing the number of pixels of the imaging device itself. However, it is generally known that the cost of the imaging device rapidly rises with the increase in the number of pixels.
The applicant of the present application proposed the technique of joining the images taken by a plurality of imaging devices or images obtained by moving a camera as described in Japanese Patent Application KOKAI Publication No. 6-141246.
However, in general when the photographed image is subjected to influence of distortion due to the optical system, the image is distorted. If the images are joined by the technique of Japanese Patent Application KOKAI Publication No. 6-141246, the composition of the overlapped parts differs between the images, and there occurs a problem object is seen double in the join image. Also, due to the displacement of the points, serving as a reference for joining, the image is detected as if it were rotated though the image is not actually rotated. As a result, there occurs a problem in which the images are not joined well.
To solve the above problems, the applicant of the present application proposed an image processing apparatus for compensating for influence of distortion comprising image correcting means for a geometrical correction as described in U.S. patent application Ser. No. 08/541,644 (filing data: Oct. 10, 1995).
The structure of the image processing apparatus is shown in FIG. 18.
In the figure, each of image input sections 1a to 1c of the image processing apparatus comprises an optical system 2, an image pickup section 3 such as CCD, and an A/D converter 4. These image input sections are arranged to Capture different portions (positions) of an object 5 to have the overlapping area.
An output signal of each image pickup section 3 is digitized by the A/D converter 4 so as to be input to each of image correcting sections 17a to 17c. Each of the image correcting sections 17a to 17c reads photographing conditions such as a focus position when a image is taken and a characteristic parameter of the optical system so as to correct the distortion of the images taken by the image input sections 1a to 1c. 
Next, in an image joining section 6, the images (serving as input signals), which are corrected by the image correcting sections 17a to 17c, are joined to be a wide-angle image as shown in FIG. 20. Then, the joined image is output to a monitor 7, a printer 8 or a storage medium 9.
The image joining section 6 is realized by the structure as shown in FIG. 19.
In this structure, the images a, b, and c are temporarily stored in a frame memory 10 respectively. Then, an amount of parallel movement S1 and an amount of rotations R1 between the adjacent images (e.g., images a and b) are obtained by a shift detector 11a. Similarly, an amount of parallel movement S2 and an amount of rotations R2 between the images b and c are obtained by a shift detector 11b. 
These amounts of parallel movement S1, S2, and amounts of rotations R1 and R2 are input to interpolation calculators 12a and 12b, together with the images read from frame memories 10b, 10c. Thereby, the images whose positional relationship are corrected can be obtained.
A coefficient setting device 13 sets coefficients Ca, Cb, and Cc of the respective images of FIG. 20 such that the adjacent images are smoothly joined to each other. The pixel value of each image is multiplied by each of coefficients Ca, Cb, Cc by a multiplier 14. Then, the overlapping portion is added by an adder 15.
FIG. 20 is a view showing the processing of the overlapping portion of the images to be joined.
The image b rotates anticlockwise against the image a. The rotation of the image b and the amount of overlapping (or amount of parallel movement) are calculated by the shift detector 11. Also, as shown in FIG. 20, the pixel value of each image is multiplied by each of coefficients Ca, Cb, Cc so as to smoothly connect the images a and b, which are overlapped with each other. In this way, the image joining section 6 outputs the image in which the plurality of images are joined with high resolution or a wide viewing angle are provided.
Regarding to extend dynamic range of the imaging device, the applicant of the present application proposed the following technique in Japanese Patent Application KOKAI Publication No. 63-232591.
Specifically, a plurality of images photographed with different exposure is synthesized so as to generate an image having a dynamic range, which is almost equal to the film.
The above technique can be realized by structuring the image joining section 6 as shown in FIG. 22.
FIG. 22 conceptually explains an example in which two images are synthesized. Even in a case of joining three or more images, the images are synthesized by the same process.
Two images a and b are added to each other at an adder 21 to be stored in the frame memory 10. A linear converting section 22 reads out data of the frame memory 10. The linear converting section 22 calculates values corresponding to R, G, B values of incident light based on a look-up table so as to be input to a matrix circuit 23. The R, G, B values obtained at this time exceed the dynamic range of an input device such as a digital still camera.
The converting table is determined from an exposure ratio, Rexp, of two images by a converting table preparation section. In the matrix circuit 23, a luminance signal value Y is obtained from R, G, B values. A luminance compression section 24 outputs a luminance value Y′ which is compressed to adjust to the output device. Then, a ratio of compressed signal to original one Y′/Y is obtained by a divider 25. The ratio Y′/Y is multiplied by outputs R, G, B of the linear converting section 22 by a multiplier 25 so as to be stored in the frame memory 16 as a joining image result.
Generally, the signal value to be output from the imaging device is saturated for a certain amount of incident light in the case of longer time exposure as shown in FIG. 21.
The value of an additional signal in which the signal of the longer exposure and that of the shorter exposure are added is changed with respect to the amount of incident light as shown by a bent line showing as an additional signal in FIG. 21. Then, a converting table preparation section 27 determines a table in which an amount of incident light I is estimated from an additional signal value S.
Generally, since the image value is expressed by 256 steps of 0 to 255, luminance Y of each pixel is compressed, for example by the following equation (1):Y′=b·ya  (1)where a is a coefficient for determining a shape of compression and b is a coefficient for determining a gain of the entire image.
If two different exposure images are synthesized to each other by the above-mentioned method, there can be obtained an image having the dynamic range almost equal to the film and can be seen well from a dark part to a bright part.
However, in the conventional technique described in U.S. patent application Ser. No. 08/541,644, photographing conditions, which are necessary for correcting distortion, and the parameter of the optical system have to be set in advance. Due to this, it is difficult to get the image having high resolution image or a wide viewing angle image and a panorama image by simply using an arbitrary photographing device which the user has.
The following will explain about the distortion with reference to FIGS. 23A, 23B, and 23C.
More specifically, the distortion is generally a geometrical deformation, which is caused in accordance with the distance from the center of a lens. If the lattice object is photographed through the optical system without distortion, the image, which is shown in FIG. 23A, can be obtained. However, if the optical system suffered from distortion, the structure, which should be photographed by straight lines, are curved as shown in FIG. 23B. Thus, if the optical system has distortion, a straight line L is curved and captured as a curved line L′ (FIG. 23C). As a result, a point Y on the straight line L is moved to a point Y′ on the curved line L′. In this case, an amount of distortion ΔR at one point on the image can be expressed by a polynomial expression (2):ΔR=A1·R3+A2·R5+  (2)where R is a distance between a center of the image and the point Y.
To get the image whose distortion is corrected, point Y′ may be moved by only an amount of distortion ΔR on the straight line connecting the center C of the image to point Y′. However coefficients A1, A2, . . . differ depending on the focal position of the optical system, it is difficult for the general user to know coefficients A1, A2, . . . correctly.
Also, coefficients A1, A2, . . . differ depending on the apparatus to be used. Due to this, when the different apparatus is used, the correction coefficient must be adjusted again.
Moreover, if images are taken by different zooming ratio, the size between adjacent images differs. Due to this, the images cannot be correctly joined to each other though the images are overlapped with each other. Also, for the object close to the user in such a case of an indoor place, the size is changed even if a photographer moves a few steps.
Moreover, there is a case in which a white balance is automatically adjusted. For example, the color tone differs depending on a case in which the object is photographed in a direction toward or away from the sun.
Due to this, when the images are joined by the above-mentioned technique, color is smoothly changed but the entire image seems unnatural one. Moreover, in the technique of the wide dynamic range, the table for estimating the amount of incident light from the additional signal must be prepared as explained in the prior art. However, as shown in FIG. 21, a point N where the inclination of the additional signal is changed varies depending on the exposure ratio Rexp. For this reason, the user must know the exposure ratio Rexp of the plurality of images to be synthesized in advance. However, in many cases, the digital still cameras on the market have the structure in which the exposure can be adjusted but the user cannot know the ratio exactly.